REPORT  DOCUMENTATION  PAGE 


Form  Approved 
OMBNO.  0704-0188 


Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources, 
gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comment  regarding  this  burden  estimates  or  any  other  aspect  of  this 
collection  erf  information,  n eluding  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  information  Operations  and  Reports.  1215  Jefferson 
Davos  Highway,  Suite  1204,  Arlington,  VA  22202-4302,  and  to  the  Office  of  Management  and  Budget,  Paperwork  Reduction  Project  (0704-0188),  Washington,  DC  20503. 


1.  AGENCY  USE  ONLY  (Leave  blank)  2.  REPORT  DATE 

3/29/01 


4.  TITLE  AND  SUBTITLE 


Stochastic  Control  Problems  in 
Moblie  Communications 


3.  REPORT  TYPEAND  DATES  COVERED 
i  exSrsrvv  GObA 

Annual  Report ;  9/1/00-5/31/01 


5.  FUNDING  NUMBERS 


DAAD1 9-00- 1-0549 


6.  AUTHOR(S) 


Paul  Dupuis  and  Harold  Kushner 


7.  PERFORMING  ORGANIZATION  NAMES(S)  AND  ADDRESS(ES) 
Division  of  Applied  Mathematics 
Brown  University 
182  George  Street 
Providence,  RI  02912 


8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 


9.  SPONSORING  /  MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

U.S.  Army  Research  Office 
P.O.Box  12211 

Research  Triangle  Park,  NC  27709-2211 


1 0.  SPONSORING  /  MONITORING 
AGENCY  REPORT  NUMBER 


A&o  A&'Ml .  i  -  A\  A 


11.  SUPPLEMENTARY  NOTES 


The  views,  opinions  and/or  findings  contained  in  this  report  are  those  of  the  author(s)  and  should  not  be  construed 
an  official  Department  of  the  Army  position,  policy  or  decision,  unless  so  designated  by  other  documentation. 


12a.  DISTRIBUTION  /  AVAILABILITY  STATEMENT 


12  b.  DISTRIBUTION  CODE 


Approved  for  public  release;  distribution  unlimited. 


13.  ABSTRACT  (Maximum  200  words) 
See  Attached 


20010503  088 


rpr  0  5  ZOO1.  |t 


15.  NUMBER  IF  PAGES 
4 


16.  PRICE  CODE 


17.  SECURITY  CLASSIFICATION  18.  SECURITY  CLASSIFICATION  19.  SECURITY  CLASSIFICATION  20.  LIMITATION  OF  ABSTRACT 
OR  REPORT  OF  THIS  PAGE  OF  ABSTRACT 

UNCLASSIFIED  UNCLASSIFIED  UNCLASSIFIED  UL 


NSN  7540-01-280-5500 


Enclosure  1 


Standard  Form  298  (Rev.  2-89) 

Prescribed  by  ANSI  Std.  239-18 
298-102 


Annual  Report  for  2000,  Army  Research  Office 
Stochastic  Control  Problems  in  Mobile  Communications 
Paul  Dupuis  and  Harold  J.  Kushner 
Applied  Mathematics  Dept. 

Brown  University,  Providence  RI  02912 
DAAD19-00-1-0549 
9/1/00-5/31/01 


1.  Numerical  Methods  of  Stochastic  Control.  The  book  [6]  contains  the  most 
comprehensive  development  of  numerical  algorithms  and  associated  convergence  proofs 
for  a  large  part  of  the  current  forms  of  stochastic  control  problems,  and  has  been  quite 
popular.  A  second  edition  was  completed.  In  addition  to  the  broad  coverage  of  the  pre¬ 
vious  edition,  it  gives  numerical  algorithms  and  proofs  for  problems  where  the  variance 
term  is  controlled,  and  for  jump-diffusions  where  the  jump  is  controlled.  The  latter  prob¬ 
lem  has  arisen  in  recent  applications  to  communications  theory.  The  standard  use  of  the 
Poisson  measure  driven  model  is  no  longer  adequate,  and  a  general  theory  is  developed. 
Additionally,  the  book  contains  a  thorough  development  for  deterministic  problems  which 
arise  in  control  and  the  calculus  of  variations,  and  includes  discontinuous  or  unbounded 
dynamical  terms,  with  applications  to  image  reconstruction,  large  deviations,  and  else¬ 
where.  The  algorithms  are  about  the  fastest  and  most  stable  available,  and  there  are 
convergence  proofs  for  all  of  them. 

2.  Routing  and  service  control  for  large  dimensional  networks.  Key  issues  in  our 
investigations  include:  (i)  the  development  of  approximate  models  that  allow  for  explicit 
(or  nearly  explicit)  construction  of  the  optimal  routing/service  policies,  and  (ii)  robustness 
and  the  ability  to  deal  with  model  perturbations.  The  problems  considered  encompass  a 
number  of  different  formulations,  each  of  which  emphasizes  different  qualitative  properties 
of  the  resulting  controlled  network.  These  include  risk-sensitive  control  and  the  control 
of  rare  events,  optimal  control  of  “fluid”  models,  and  optimally  robust  control  of  such 
models.  “Fluid”  models  are  approximate  models  that  are  obtained  under  a  law  of  large 
numbers  scaling.  In  all  cases  the  system  model  is  constrained,  and  in  most  problems 
this  is  a  dominant  feature.  To  handle  the  constraints,  we  model  the  state  dynamics 
of  the  approximate  (or  limit)  models  in  terms  of  an  appropriate  Skorokhod  Problem 
and  corresponding  constrained  ordinary  or  stochastic  differential  equations.  The  different 
problem  setups  (e.g.,  control  of  rare  events  versus  control  of  fluid  models)  lead  to  problems 
of  the  same  basic  form,  which  is  a  Skorokhod  Problems  with  (relatively)  simple  dynamics 
and  simple  cost  structures. 

Although  the  general  theory  of  optimal  control  for  such  network  approximations  is  still 
in  the  early  stages  of  development,  there  are  already  a  number  of  interesting  and  important 
problem  classes  for  which  explicit  solutions  have  been  found.  These  classes  of  problems 
provide  analogues  in  the  setting  of  constrained  processes  for  the  familiar  linear/quadratic 
problems  from  the  theory  of  unconstrained  processes.  As  such,  they  may  eventually  play 
as  important  a  role  in  the  developing  theory  for  problems  with  constraints.  Examples 
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include  references  [4,  5]  The  collection  of  problems  with  explicit  solutions  will  continue 
to  expand,  as  we  become  more  familiar  with  the  geometries  and  cost  structures  that  are 
natural  for  constrained  processes  (from  the  point  of  view  of  explicitly  solvable  problems), 
and  the  new  techniques  which  allow  for  their  solution.  Reference  [4]  deals  with  a  controlled 
constrained  ordinary  differential  equation,  and  with  a  cost  that  depends  only  on  the 
control.  Such  problems  occur  in  the  (ordinary)  control  of  fluid  models,  such  as  control  of 
a  network  with  the  objective  of  reducing  backlogs  in  minimum  time.  Our  analysis  gives  an 
explicit  finite-dimensional  representation  of  the  value  function,  and  identifies  all  optimal 
controls.  Reference  [5]  considers  the  problem  of  characterizing  the  manner  in  which  rare 
events  occur  in  networks  under  heavy  traffic.  It  shows  how  time  reversal  arguments  can 
be  applied  to  rewrite  the  control  problem  defined  by  a  straightforward  large  deviations- 
analysis  into  the  form  of  [2],  and  then  shows  how  explicit  finite-dimensional  solutions  can 
be  obtained  for  some  interesting  classes  of  problems  (in  arbitrary  dimension).  A  three 
dimensional  example  is  worked  out  in  detail  for  illustrative  purposes.  Reference  [2]  deals 
with  related  problems  of  stability  of  constrained  processes. 

3.  Polling,  and  applications  to  wireless,  controlled  jumps,  and  stability.  Con¬ 
sider  a  queueing  system  with  many  queues,  and  one  server  which  allocates  its  time  among 
the  queues  to  minimize  some  cost.  Usually,  it  is  assumed  that  the  queues  are  always 
available,  and  the  server  can  allocate  at  will.  In  modern  applications  to  wireless  or  mil¬ 
itary  systems,  there  are  randomly  occurring  periods  where  a  source  is  unavailable,  and 
this  must  be  accounted  for  in  the  polling  policy.  The  possibility  of  such  “vacations”  com¬ 
plicates  the  problem  enormously,  and  there  is  little  literature  on  the  subject,  despite  the 
vast  amount  on  polling.  Again,  due  to  the  complexity  of  the  basic  problem,  we  analyze 
[1]  it  in  the  heavy  traffic  regime  where  the  server  has  little  idle  time  over  the  average 
requirements.  The  scaled  total  workload  process  converges  to  a  controlled  limit  diffusion 
process  with  jumps,  which  are  due  to  the  effects  of  the  vacations.  The  control  enters  the 
dynamics  only  via  its  value  just  before  a  vacation  begins;  hence  it  is  only  via  the  the 
random  jump  value  that  the  control  affects  the  dynamics.  This  type  of  model  has  not  re¬ 
ceived  any  attention.  The  limit  problem  can  be  represented  in  terms  of  a  one  dimensional 
reflected  jump-diffusion,  no  matter  the  number  of  queues,  which  greatly  simplifies  the 
limit  and  enhances  the  value  of  the  approach.  Many  results  of  considerably  broader  use 
were  obtained,  such  as  averaging  principles  which  assure  the  validity  of  a  fluid  approxi¬ 
mation  during  the  vacation.  Explicit  analytical  and  numerical  solutions  were  obtained  in 
some  simple  but  important  cases,  and  they  are  not  intuitive,  owing  to  the  anticipation  of 
the  effects  of  the  vacations.  Stability  and  unreliable  channels  were  treated,  under  general 
conditions  on  the  data. 

4.  Heavy  traffic  modeling  for  communications  systems.  We  have  been  writing  a 
comprehensive  tratise  on  the  use  of  heavy  traffic  methods  for  the  simplification  and  anal¬ 
ysis  of  complex  communications  systems  and  related  queueing  type  networks.  The  aim  of 
the  book  is  the  development  of  the  heavy  traffic  approach  to  the  modeling  and  analysis 
of  queueing  networks,  both  controlled  and  uncontrolled,  and  many  applications  to  com¬ 
puter,  communications,  and  manufacturing  systems.  The  methods  exploit  the  multiscale 
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structure  of  the  physical  problem  to  get  approximating  models  that  have  the  form  of  re¬ 
flected  diffusion  processes,  either  controlled  or  uncontrolled.  These  approximating  models 
have  the  basic  structure  of  the  original  problem,  but  are  significantly  simpler.  Much  of 
inessential  detail  is  eliminated  (or  “averaged  out”).  They  greatly  simplify  analysis,  de¬ 
sign,  and  optimization  and  yield  good  approximations  to  problems  that  would  otherwise 
be  intractable,  under  broad  conditions. 

One  is  concerned  with  rather  general  inter  arrival  and  service  intervals,  complicated 
network  topologies  and  routing  schemes,  control  or  optimal  control  mechanisms,  effec¬ 
tive  numerical  methods,  stability  questions,  state  dependence,  service  interruptions,  non- 
Markov  models,  and  other  complications  such  a  those  due  to  finite  or  shared  buffers, 
feedback,  blocking,  or  the  balancing  of  the  demands  or  the  scheduling  of  several  different 
job  classes.  Correlations  in  the  service  times  for  the  same  job  as  it  proceeds  from  processor 
to  processor  and  other  forms  of  correlation  are  also  current  issues,  as  are  randomly  vary¬ 
ing  parameters,  perhaps  due  to  varying  environmental  factors.  The  demands  of  modern 
high-speed  and  high-performance  communications  pose  very  serious  challenges,  since  they 
must  operate  with  very  small  error  rates.  Although  there  is  considerable  effort  devoted 
to  these  problems,  a  great  many  of  the  most  significant  remain  intractable.  When  one 
wishes  to  apply  optimal  control  methods,  the  exact  physical  model  is  often  much  too 
complicated  for  an  exact  solution  to  be  obtained. 

Although  one  would  like  to  solve  the  problems  without  any  type  of  approximation, 
asymptotic  methods  are  often  the  best  solution  (or  even  the  only  hope).  We  are  concerned 
with  the  approximation  of  controlled  or  uncontrolled  queueing  networks  by  reflected  dif¬ 
fusion  or  jump-diffusion  processes.  Diffusion  approximations  have  long  been  an  effective 
tool  for  simplifying  or  approximating  physical  or  biological  problems  The  reflected  diffu¬ 
sion  approximation  simplifies  the  queueing  network  problem  in  many  ways.  It  is  Marko¬ 
vian  and  lower  (often  much  lower)  dimensional,  numerical  methods  are  available,  and  the 
structure  is  exhibited  more  clearly.  Even  formal  analyses  of  complex  problems  that  are 
guided  by  the  insights  and  approximations  of  heavy  traffic  analysis  can  lead  to  impressive 
results. 

5.  Analysis  and  Control  of  Mobile  Communications  with  Time  Varying  Chan¬ 
nels  in  Heavy  Traffic.  Consider  a  system  with  a  fixed  number  ( K )  of  remote  units  and 
a  single  base  transmitter  with  time  varying  (and  perhaps  correlated)  connecting  channels. 
Data  to  be  transmitted  to  the  remote  units  arrives  according  to  some  random  process  and 
is  queued  according  to  its  destination.  The  uplink  is  treated,  for  specificity.  Power  is  to 
be  allocated  to  the  K  channels  in  a  time  varying  way.  The  modeling  and  control  problem 
can  be  quite  difficult.  The  channel  time  variations  (fading)  are  fast  and  the  bandwidth 
and  data  arrival  rate  are  high.  Owing  to  the  complexity  of  the  physical  problem  and  the 
high  speed  of  both  the  fading  and  arrival  and  service  rates,  an  asymptotic  or  averaging 
method  is  promising.  A  heavy  traffic  analysis  was  done  in  [3] . 

By  heavy  traffic,  we  mean  that  on  the  average  there  is  little  server  idle  time  and  little 
spare  power.  Heavy  traffic  analysis  has  been  very  helpful  in  simplifying  analysis  of  both 
controlled  and  uncontrolled  problems  in  queueing  and  communications  networks.  It  tends 
to  eliminate  unessential  detail  and  focus  on  the  fundamental  issues  of  scaling  and  para- 
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metric  dependencies.  A  variety  of  models  are  considered.  The  basic  model  assumes  that 
the  channel  state  is  known  or  can  be  well  estimated  and  that  given  the  channel  state  there 
is  a  well  defined  rate  of  transmission  per  unit  power.  Then  convergence  of  the  controlled 
scaled  queue  lengths  is  shown.  The  scaling  is  different  from  the  usual  in  heavy  traffic 
work,  and  the  limit  Wiener  process  depends  only  on  the  channel  state  process  and  not  on 
the  data  arrival  process.  The  appropriate  orders  of  reserve  power  and  buffer  size  is  given 
as  well  as  suggested  policies.  The  limit  or  approximating  process  is  a  controlled  reflected 
diffusion  which  is  much  simpler  than  the  original  problem  and  facilitates  understanding 
parametric  dependencies,  solutions  and  stability.  There  are  new  issues  in  computing  the 
reflection  direction.  Additionally,  a  more  general  model  is  considered,  where  knowledge 
of  the  channel- can  be  subject  to  random  errors,  delays  and  other  perturbations  as  well  as 
power  scheduling  constraints.  The  methods  of  proof  and  limit  forms  (hence  the  limit  or 
approximating  control  problems)  are  similar. 

The  analysis  and  results  do  not  depend  on  the  details  of  coding,  modulation  or  de¬ 
tection:  These  are  represented  by  the  transmission  rate  processes.  This  procedure  allows 
considerable  generality  as  well  as  focusing  on  what  is  essential. 
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